Detecting Pilot's Engagement Using fNIRS Connectivity Features in an Automated vs. Manual Landing Scenario

Monitoring pilot's mental states is a relevant approach to mitigate human error and enhance human machine interaction. A promising brain imaging technique to perform such a continuous measure of human mental state under ecological settings is Functional Near-InfraRed Spectroscopy (fNIRS). However, to our knowledge no study has yet assessed the potential of fNIRS connectivity metrics as long as passive Brain Computer Interfaces (BCI) are concerned. Therefore, we designed an experimental scenario in a realistic simulator in which 12 pilots had to perform landings under two contrasted levels of engagement (manual vs. automated). The collected data were used to benchmark the performance of classical oxygenation features (i.e., Average, Peak, Variance, Skewness, Kurtosis, Area Under the Curve, and Slope) and connectivity features (i.e., Covariance, Pearson's, and Spearman's Correlation, Spectral Coherence, and Wavelet Coherence) to discriminate these two landing conditions. Classification performance was obtained by using a shrinkage Linear Discriminant Analysis (sLDA) and a stratified cross validation using each feature alone or by combining them. Our findings disclosed that the connectivity features performed significantly better than the classical concentration metrics with a higher accuracy for the wavelet coherence (average: 65.3/59.9 %, min: 45.3/45.0, max: 80.5/74.7 computed for HbO/HbR signals respectively). A maximum classification performance was obtained by combining the area under the curve with the wavelet coherence (average: 66.9/61.6 %, min: 57.3/44.8, max: 80.0/81.3 computed for HbO/HbR signals respectively). In a general manner all connectivity measures allowed an efficient classification when computed over HbO signals. Those promising results provide methodological cues for further implementation of fNIRS-based passive BCIs

Spectral EEG-based classification for operator dyads workload and cooperation level estimation

There is a growing momentum to design online tools to measure mental workload for neuroergonomic purposes. Most of the research focuses on the monitoring of a single human operator. However, in real-life situations, human operators work in cooperation to optimize safety and performance. This is particularly the case in aviation whereby crews are composed of a pilot flying and a pilot monitoring. The motivation of this study is to evaluate the possibility to apply an hyperscanning approach to estimate the mental workload of crews composed of two operators. We designed an experimental protocol in which ten crews (i.e. 20 subjects) had to perform a modified version of the NASA MATBII during 8 five-minute blocks (i.e. 4 mental workload level configurations * 2 cooperation v. non cooperation conditions). Mental workload and cooperation level were classified using a traditional passive brain-computer interface pipeline that includes a spatial filtering step on frequency features. Our results disclosed that all mental states’ estimations were significantly above chance level. Intra-subject classification accuracy for mental workload (2 classes) was 63% for the pilot flying and 58% for the pilot monitoring. As for cooperation level, the binary classification reached 57% for the pilot flying and 60% for the pilot monitoring. Regarding the team, intra-team classification accuracy of the workload configuration of the team (4-class) reached 35%. As for the team cooperation level, the binary classifier reached 60% of accuracy. The results are discussed in terms of hyperscanning applications

Pre-stimulus antero-posterior EEG connectivity predicts performance in a UAV monitoring task

Long monitoring tasks without regular actions, are becoming increasingly common from aircraft pilots to train conductors as these systems grow more automated. These task contexts are challenging for the human operator because they require inputs at irregular and highly interspaced moments even though these actions are often critical. It has been shown that such conditions lead to divided and distracted attentional states which in turn reduce the processing of external stimuli (e.g. alarms) and may lead to miss critical events. In this study we explored to which extent it is possible to predict an operator’s behavioural performance in a Unmanned Aerial Vehicle (UAV) monitoring task using electroencephalographic (EEG) activity. More specifically we investigated the relevance of large-scale EEG connectivity for performance prediction by correlating relative coherence with reaction times (RT). We show that long-range EEG relative coherence, i.e. between occipital and frontal electrodes, is significantly correlated with RT and that different frequency bands exhibit opposite effects. More specifically we observed that coherence between occipital and frontal electrodes was: negatively correlated with RT at 6Hz (theta band), more coherence leading to better performance, and positively correlated with RT at 8Hz (lower alpha band), more coherence leading to worse performance. Our results suggest that EEG connectivity measures could be useful in predicting an operator’s attentional state and her/his performances in ecological settings. Hence these features could potentially be used in a neuro-adaptive interface to improve operator-system interaction and safety in critical systems

Physiological synchrony revealed by delayed coincidence count: Application to a cooperative complex environment

Synchrony at the physiological level is an objective measure that can be used to investigate cooperation between human agents. This physiological synchrony has been experimentally observed in different dyadic contexts through measures of the autonomous system such as cardiac measures. Various metrics were used to characterize synchrony between participants such as cross-correlation, weighted coherence or cross recurrence quantification analysis (CRQA) and with a wide variety of paradigms. We propose the delayed coincidence count as a new method for assessing cardiac synchrony. Delayed coincidence count has already been used to characterize synchrony in firing neurons populations. While being straightforward and computationally light, this method has already been formally proven to be statistically robust. A complex dynamic micro-world was designed with two difficulty levels and two cooperation conditions. Forty participants, i.e. 20 teams, voluntarily underwent the experiment. The delayed coincidence count method (with a coincidence threshold delta of 20 ms) revealed a significant synchrony (p < .01) during the cooperative and high difficulty condition only, while the other methods did not. The results are interpreted in terms of interaction intensity in accordance with recent literature

Monitoring pilot’s cognitive fatigue with engagement features in simulated and actual flight conditions using an hybrid fNIRS-EEG passive BCI

There is growing interest for implementing tools to monitor cognitive performance in naturalistic environments. Recent technological progress has allowed the development of new generations of brain imaging systems such as dry electrodes electroencephalography (EEG) and functional near infrared spec- troscopy (fNIRS) to investigate cortical activity in a variety of human tasks out of the laboratory. These highly portable brain imaging devices offer interesting prospects to implement passive brain computer interfaces (pBCI) and neuroadaptive technology. We developed a fNIRS-EEG based pBCI to monitor cognitive fatigue using engagement related features (EEG engagement ratio and wavelet coherence fNIRS based metrics). This mental state is known to impair cognitive performance and can jeopardize flight safety. In this preliminary study, four participants were asked to perform four identical traffic patterns along with a secondary auditory task in a flight simulator and in an actual light aircraft. The two first traffic patterns were considered as the low cognitive fatigue class, whereas the two last traffic patterns were considered as the high cognitive fatigue class. As expected, the pilots missed more auditory targets in the second part than in the first part of the experiment. Classification accuracy reached 87.2% in the flight simulator condition and 87.6% in the actual flight conditions when combining the two modalities. This study demonstrates that fNIRS and EEG-based pBCIs can monitor mental states in operational and noisy environments